Feature 1: AI Confidence & Uncertainty Estimation#152
Conversation
|
Someone is attempting to deploy a commit to the karan3431's projects Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughFusion inference now computes a system confidence score and uncertainty flag via a new calculateConfidence helper, exposing confidenceScore and uncertainFlag in predict results. Backend derives uncertain_flag from stored confidence_score. Dashboard renders a new uncertainty warning banner based on this flag. ChangesConfidence and Uncertainty Feature
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant Dashboard as AnalysisDashboard
participant Backend as backend/main.py
participant Fusion as fusionInference.js
Fusion->>Fusion: calculateConfidence(bodyProbs, eyeProbs, gillProbs)
Fusion-->>Fusion: confidenceScore, uncertainFlag
Backend->>Backend: _row_to_payload derives uncertain_flag
User->>Dashboard: view scan result
Dashboard->>Backend: fetch scan payload (uncertain_flag)
Backend-->>Dashboard: scan payload with uncertain_flag
Dashboard->>Dashboard: compute isUncertain
Dashboard-->>User: render Model Uncertainty Warning banner
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/main.py`:
- Line 302: The `uncertain_flag` calculation in `backend/main.py` is treating
valid falsy confidence values as missing, so update the row payload logic to use
an explicit `None` check instead of `or 1.0`. In the same code path that builds
the response with `confidence_score`, make `uncertain_flag` derive from the
actual stored value from `row.get("confidence_score")` (using the same default
semantics as the nearby confidence field) so a real `0.0` stays `0.0` and
evaluates as uncertain, while missing values use one consistent default.
In `@src/fusionInference.js`:
- Around line 288-296: Fix the downstream NaN propagation in the fusion return
path by ensuring calculateConfidence always produces a valid numeric confidence
before assigning systemConfidence, confidenceScore, and uncertainFlag in
fusionInference.js. Also replace the inline 0.70 uncertainty check in the fusion
logic with a shared named constant (for example UNCERTAINTY_THRESHOLD) placed
with the existing THRESHOLD_* constants so this value stays aligned with
backend/main.py and can be reused consistently.
- Around line 259-268: The confidence calculation in calculateConfidence is
indexing eyeProbs and gillProbs as if they were 4-element arrays, but
extractEyeScore and extractGillScore supply 2-element values, so the current
eye/gill math can produce undefined and NaN. Update calculateConfidence to use
the correct 0/1 indices for the arrays it actually receives, or change the
callers to pass the full logits consistently, and make sure the eye/gill
confidence paths still feed uncertainFlag correctly.
- Around line 393-394: fuseFromLogits() is returning a result shape that is
missing confidenceScore and uncertainFlag compared with predict(). Update the
fuseFromLogits return object in fusionInference.js to include those fields, and
also extend FusionResult in fusionInference.d.ts so the helper’s type matches
the main inference path.
In `@src/pages/AnalysisDashboard.tsx`:
- Around line 117-122: The Model Uncertainty Warning title and description in
AnalysisDashboard are hardcoded English strings instead of using the existing
translation flow. Replace these literals with t('dashboard.*') keys in the same
component, and add matching entries to every locale file so the warning is fully
localized like the rest of the dashboard copy.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 75c5c89e-61db-40e8-b507-3bcb3c8f5256
📒 Files selected for processing (3)
backend/main.pysrc/fusionInference.jssrc/pages/AnalysisDashboard.tsx
| "classification": "FRESH" if is_fresh else "SPOILED", | ||
| "is_fresh": is_fresh, | ||
| "uncertain_flag": False, | ||
| "uncertain_flag": (row.get("confidence_score") or 1.0) < 0.70, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Logic bug: or 1.0 mishandles a stored 0.0 confidence and contradicts the Line 299 default.
(row.get("confidence_score") or 1.0) short-circuits on any falsy value, so a legitimately stored confidence of 0.0 (maximally uncertain) is replaced with 1.0, yielding uncertain_flag = False — the opposite of intended.
It also disagrees with Line 299, where a missing confidence_score defaults to 0 (→ 0% confidence). So for a row with no score, the payload reports confidence: 0.0 yet uncertain_flag: False. Use an explicit None check and a consistent default.
🐛 Proposed fix
- "uncertain_flag": (row.get("confidence_score") or 1.0) < 0.70,
+ "uncertain_flag": (
+ row["confidence_score"] < 0.70
+ if row.get("confidence_score") is not None
+ else True
+ ),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "uncertain_flag": (row.get("confidence_score") or 1.0) < 0.70, | |
| "uncertain_flag": ( | |
| row["confidence_score"] < 0.70 | |
| if row.get("confidence_score") is not None | |
| else True | |
| ), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/main.py` at line 302, The `uncertain_flag` calculation in
`backend/main.py` is treating valid falsy confidence values as missing, so
update the row payload logic to use an explicit `None` check instead of `or
1.0`. In the same code path that builds the response with `confidence_score`,
make `uncertain_flag` derive from the actual stored value from
`row.get("confidence_score")` (using the same default semantics as the nearby
confidence field) so a real `0.0` stays `0.0` and evaluates as uncertain, while
missing values use one consistent default.
| function calculateConfidence(bodyProbs, eyeProbs, gillProbs) { | ||
| const bodyConf = Math.max(...bodyProbs); | ||
| const eyeSubSum = (eyeProbs[0] + eyeProbs[2]) > 0 ? (eyeProbs[0] + eyeProbs[2]) : 1e-7; | ||
| const gillSubSum = (gillProbs[1] + gillProbs[3]) > 0 ? (gillProbs[1] + gillProbs[3]) : 1e-7; | ||
|
|
||
| const eyeConf = Math.max(eyeProbs[0] / eyeSubSum, eyeProbs[2] / eyeSubSum); | ||
| const gillConf = Math.max(gillProbs[1] / gillSubSum, gillProbs[3] / gillSubSum); | ||
|
|
||
| return (0.5 * bodyConf) + (0.25 * eyeConf) + (0.25 * gillConf); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
ast-grep run --pattern 'function extractEyeScore($_, $_) { $$$ }' --lang javascript src/fusionInference.js
ast-grep run --pattern 'function extractGillScore($_, $_) { $$$ }' --lang javascript src/fusionInference.js
rg -nP 'processAndFuse\s*\(' src/fusionInference.js -C2Repository: jpdevhub/FreshScanAi
Length of output: 2058
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '180,310p' src/fusionInference.js | cat -nRepository: jpdevhub/FreshScanAi
Length of output: 5744
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
text = Path('src/fusionInference.js').read_text()
for name in ['extractBodyScore', 'extractEyeScore', 'extractGillScore', 'calculateConfidence', 'processAndFuse']:
idx = text.find(f'function {name}')
print(f'\n### {name} @ {idx}')
if idx != -1:
start = max(0, idx-300)
end = min(len(text), idx+1200)
print(text[start:end])
PYRepository: jpdevhub/FreshScanAi
Length of output: 6306
Fix the eye/gill confidence indices
calculateConfidence treats eyeProbs/gillProbs as 4-element vectors, but extractEyeScore() and extractGillScore() pass 2-element arrays. eyeProbs[2] and gillProbs[3] are undefined, so the confidence collapses to NaN and uncertainFlag never trips.
Use indices 0/1 here, or pass the raw 4-class logits instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/fusionInference.js` around lines 259 - 268, The confidence calculation in
calculateConfidence is indexing eyeProbs and gillProbs as if they were 4-element
arrays, but extractEyeScore and extractGillScore supply 2-element values, so the
current eye/gill math can produce undefined and NaN. Update calculateConfidence
to use the correct 0/1 indices for the arrays it actually receives, or change
the callers to pass the full logits consistently, and make sure the eye/gill
confidence paths still feed uncertainFlag correctly.
| confidenceScore: fusion.confidenceScore, | ||
| uncertainFlag: fusion.uncertainFlag, |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -nP '(confidenceScore|uncertainFlag|confidence_score|uncertain_prediction_flag|uncertain_flag)' -g '!**/node_modules/**' -C2
ast-grep run --pattern 'export function fuseFromLogits($$$) { $$$ }' --lang javascript src/fusionInference.jsRepository: jpdevhub/FreshScanAi
Length of output: 158
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== src/fusionInference.js outline ==\n'
ast-grep outline src/fusionInference.js --view expanded || true
printf '\n== Relevant sections in src/fusionInference.js ==\n'
nl -ba src/fusionInference.js | sed -n '340,520p'
printf '\n== Search for confidence/uncertain fields ==\n'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/.git/**' \
'(confidenceScore|uncertainFlag|confidence_score|uncertain_prediction_flag|uncertain_flag)' \
.
printf '\n== backend/main.py relevant section ==\n'
if [ -f backend/main.py ]; then
nl -ba backend/main.py | sed -n '220,310p'
else
printf 'backend/main.py not found\n'
fiRepository: jpdevhub/FreshScanAi
Length of output: 1817
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== src/fusionInference.js around predict() ==\n'
sed -n '354,485p' src/fusionInference.js | cat -n
printf '\n== backend/main.py around fusion consumer ==\n'
if [ -f backend/main.py ]; then
sed -n '240,285p' backend/main.py | cat -n
else
printf 'backend/main.py not found\n'
fi
printf '\n== Search for fuseFromLogits call sites ==\n'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/.git/**' 'fuseFromLogits\(' .
printf '\n== Search for predict() result key usage ==\n'
rg -n --hidden --glob '!**/node_modules/**' --glob '!**/.git/**' '(confidenceScore|uncertainFlag|confidence_score|uncertain_prediction_flag)' .Repository: jpdevhub/FreshScanAi
Length of output: 10985
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== FusionResult type/shape ==\n'
sed -n '1,140p' src/fusionInference.d.ts | cat -n
printf '\n== Fusion math return shape around processAndFuse ==\n'
sed -n '250,320p' src/fusionInference.js | cat -n
printf '\n== fuseFromLogits smoke test usage ==\n'
sed -n '480,520p' src/fusionInference.js | cat -nRepository: jpdevhub/FreshScanAi
Length of output: 6871
Keep fuseFromLogits() aligned with predict() — it still omits confidenceScore and uncertainFlag, and src/fusionInference.d.ts doesn’t declare them either, so callers of the helper get a different result shape than the main inference path. Add those fields to both the JS return object and FusionResult.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/fusionInference.js` around lines 393 - 394, fuseFromLogits() is returning
a result shape that is missing confidenceScore and uncertainFlag compared with
predict(). Update the fuseFromLogits return object in fusionInference.js to
include those fields, and also extend FusionResult in fusionInference.d.ts so
the helper’s type matches the main inference path.
saidai-bhuvanesh
force-pushed
the
feat/confidence-uncertainty
branch
from
6d288bf to
8f7a68c
Compare
1 participant
🔗 Upstream Issue Connection
Closes #151
This Pull Request is officially linked to and resolves Issue #151 (Feature 1: AI Confidence & Uncertainty Estimation) in the upstream repository.
Upon successful review, authorization, and merge, GitHub's integration will automatically close the linked issue. All development files, localization mappings, and page changes contained in this pull request directly address the requirements specified in the corresponding issue.
This PR implements Feature 1: AI Confidence & Uncertainty Estimation.
Changes Made
Display prediction confidence, warn when model is uncertain, suggest rescanning if confidence is low, and show uncertainty warning message on dashboard.
Summary by CodeRabbit
New Features
Bug Fixes